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NOMENCLATURE 


FTy. Fourier transform in x 

U( ) Fourier transform of g( ) 

l{x,t) moving line 

Igixj) strobed moving line 

l^{x,t) staircased moving line 

r velocity, deg/sec 

s(/) time sampling function 

t time, sec 

u spatial frequency, cycles/deg 

III spatial acuity, cycles/deg 

Uq image spatial band limit, cycles/deg 

u{t) unit pulse 

>v temporal frequency, Hz 

vv^ critical sampling frequency, Hz 

vvy' recording temporal band limit, Hz 

wj temporal acuity, Hz 

sampling frequency, Hz 
X position, deg 

z(x,t) stair function 

At temporal sampling interval, sec 

Ax spatial sampling interval, deg 

5(r) unit impulse 

* convolution 
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THE WINDOW OF VISIBILITY: A PSYCHOPHYSICAL THEORY OF FIDELITY 


IN TIME-SAMPLED VISUAL MOTION DISPLAYS 
Andrew B. Watson, Albert Ahumada, Jr,, and Joyce E. Farrell* 

Ames Research Center 

SUMMARY 

Many visual displays, such as movies and television, rely upon sampling in the time domain. The spatio- 
temporal frequency spectra for some simple moving images are derived and illustrations of how these spec- 
tra are altered by sampling in the time domain are provided. A simple model of the human perceiver which 
predicts the critical sample rate required to render sampled and continuous moving images indistinguishable 
is constructed. The rate is shown to depend upon the spatial and temporal acuity of the observer, and upon 
the velocity and spatial frequency content of the image. Several predictions of this model are tested and 
confirmed. The model is offered as an explanation of many of the phenomena known di% apparent motion. 
Finally, the implications of the model for computer-generated imagery are discussed. 

INTRODUCTION 


A film of an object in motion presents us with a sequence of static views, yet we usually see the object 
moving smoothly across the screen. This and other varieties of apparent motion have fascinated and chal- 
lenged psychologists for over a century (Exner, 1875; Braddick, 1974: Kolers, 1972; Morgan, 1979, 1980a, 
1980b). It has also become a problem of considerable applied as well as theoretical interest with the advent 
of computer-generated displays. The applied question is: how often must we present a new view for the 
stroboscopic display to faithfully simulate smooth motion? The theoretical question may be stated: how can 
a sequence of stationary images simulate a smooth motion, and why is this particular strobe rate required? 

Previous attempts to answer these questions have suffered in part from lack of an objective measure of 
how well the stroboscopic display simulates a continuous display. The strictest possible criterion for fidelity 
is considered here: the ability of a human observer to visually discriminate, by whatever means, between 
stroboscopic and continuous displays. This permits us to detennine the conditions under which stroboscopic 
and continuous motion are visually identical. The perceptual identity of continuous and stroboscopic dis- 
plays is then explained in terms of the known spatial and temporal properties of the human visual system. 

This explanation could take either of two forms. An examination of either the stimuli and visual 
mechanisms in terms of their representation in space and time, or of these mechanisms in spatial and tem- 
poral frequencies could be made. Though the two explanations are equivalent, the explanation is simpler in 
the frequency domain. 


TIME-SAMPLED MOVING IMAGES 


In a stroboscopic display the stimulus is a time-sampled version of a corresponding real motion. For 
example, to create the appearance of a vertical line with unit contrast^ moving smoothly to the right at a 
velocity r, we present a succession of brief views of the line, each following the other by an interval of time 


*Current address: Psychology Department, New York University, New York, NY 10003. 

^Contrast is defined as the luminance of the line, less the background luminance, divided by the background luminance. 
In a unit contrast line, the line has a luminance of two units, the background one unit. 



A/^, and each displaced to the right by a distance Ax = r At. The sampling frequency (w^) is the inverse of the 
time between presentations (w^ = \/At). In addition, each sample is presented with contrast At, so that the 
time-average contrasts of smooth and sampled versions are equated. Figure 1(a) plots the position of the 
smoothly moving line as a function of time; the graph is a line through the origin with slope r. Figure 1(b) 
shows the corresponding graph for the sampled version; it is a sequence of points lying along a line through 
the origin with slope r. 


CONTRAST DISTRIBUTION OF CONTINUOUS MOTION 


Figures 1(a) through 1(d) show the contrast distributions and frequency spectra for smooth and strobo- 
scopic motion. Figures 1(a) and (b) may also be viewed as distributions of contrast over space and time in 
the smooth and sampled images, respectively. The points and lines in the graphs should then be regarded as 
impulses and line impulses projecting out from the page. For example, the contrast distribution for the 
smoothly moving line (fig. 1(a)) is 


l(xj) = d(x - rt) (1) 

where l(x,t) specifies the contrast in the line at each point in horizontal space x and time and where 5( ) is 
the impulse function. The function l(xj) is a line impulse in the xj space. 


CONTRAST DISTRIBUTION OF STROBOSCOPIC MOTION 


The stroboscopic presentation is accomplished by presenting the line brietly every A^ sec at a contrast 
of At. This amounts to multiplying by a sampling function 


sit) = At Y. O) 

n~^oo 

This has the effect of exposing the line only at times that are integral multiples of At. Then the stroboscopic 
moving line is given by l^{xj) 


oo 

= l{xj)s{t) - A/6(a* - rt) ^ b{t - n AO (3) 

n--oo 

This contrast distribution for sampled motion is shown in figure 1(b). It is a sequence of impulses, separated 
by Ax in the a dimension and At in the t dimension. Notice that each impulse is multiplied by At, so that 
the contrast per unit time and space is the same in smooth and sampled images. 


FREQUFNCY SPECTRUM OF CONTINUOUS MOTION 


These distributions may be Fourier-transfonned to provide a description of the spatial and temporal 
frequency components that make up each stimulus. The transform of the smoothly moving line, L(u,\v), is 
easily determined by application of the shift theorem 




TEMPORAL FREQUENCY, Hz 


Figure 1.— Graphs and spectra of smooth and sampled lines. Points and lines should be viewed as impulses and line impulses 
projecting out from the page, (a) The distribution of contrast over space and time of a line moving smoothly to the right at 
velocity r deg/sec. The distribution is 6(x - rt) where 6 is the impulse function, (b) Contrast distribution of a sampled ver- 
sion of the moving line. The points indicate the times and positions at which the samples are presented. The distribution is 


Atd(x “ rt) ^ ] 5(r - // A/), (c) The spatiotemporal frequency spectrum of the smoothly moving line. To create a 

n--oo 

smoothly moving line from sinusoidal components, we require all spatial frequencies, and their respective temporal frequen- 
cies must increase in proportion to the spatial frequency. The spectrum is 5(w + ur) where w is temporal frequency in Hz 
and u is spatial frequency in cycles per deg. (d) The spectrum of the time-sampled moving line is identical to the spectrum 

oo 

in (c) except for the addition of parallel replicas at intervals of The spectrum is ^ ^ 5(\v + ur- nw^). 

tl=~oo 
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( 4 ) 


L(m,w) = F7'^,{/(x./)( 

^FTf e-“7rrm 

= 6(w + ru) 

where FT indicates the Fourier transform and where u is horizontal spatial frequency in cycles/deg and w is 
temporal frequency in Hz. Figure 1(c) shows that this spectrum is a line impulse passing through the origin 
with a slope of r * , 

An intuitive derivation of this result is revealed in the construction of a stationary line from sinusoidal 
components. Figure 2 illustrates how this is done by adding together an infinity of sinusoids, all with peaks 
coinciding at the position of the desired line. At that position, the many sinusoids add up to form the 
impulse; at all other points their values sum to zero. To make this line move, each sinusoid must be trans- 
lated at the same velocity, so that the peaks continue to coincide at the location of the line. But the tem- 
poral frequency of a sinusoid in motion is equal to the product of its spatial frequency and its velocity 
(vv - ur), so the temporal frequency of each sinusoid making up the line must increase in proportion to 
spatial frequency, with a proportionality constant of r (see fig. 1(c)). 


FREQUENCY SPECTRUM OF STROBOSCOPIC MOTION 


To find the transform of the sampled motion, we use the convolution theorem 

= j\s(t)l(x.r)\ 

= S{w)*L{u,w) 

oo 

= S(w + ru)* ^ 5{w-n/At) 

n--oo 

oo 

= 5(H’ + rw-mv^) (5) 

n=-oo 

This transionnation is shown in figure 1(d). It is the same as that for smooth motion except for the addition 
of parallel replicas at intervals of vv^ Hz. 


WINDOW OF VISIBILITY 


It has been known since 1956 (Shade, 1956) that the human eye is not equally sensitive to contrast 
variation at all spatial frequencies, and that sinusoidal variations above a critical spatial frequency are invisi- 
ble. Similarly, de Lange (1954) showed that temporal contrast lluctuations more rapid than a critical 
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(a) 



(b) 


Figure 2.— Line constructed from sinusoids, (a) Five sinusoids whose peaks superimpose at a point, (b) The result of adding the 
five sinusoids together and dividing by five. Where the peaks coincide, the sum shows a bump. The addition of more 
sinusoids would make the bump narrower and the other ripples smaller. The sum of an infinity of sinusoids is an impulse. 


temporal frequency are not seen. These limits to spatial and temporal frequency sensitivity will be called 
uj and wj, respectively. These two limits have been shown to be relatively independent of each other: the 
spatial limit does not depend much upon the temporal frequency of the stimulus, and vice versa (Robson, 
1966; Koenderink and van Doom, 1979). This permits us to approximate the limits of human visual sensitiv- 
ity to spatial and temporal frequencies by a window of visibility (fig. 3). Components that lie within the 
window may be more or less visible, but those that lie outside the window are invisible. This description of 
spatiotemporal contrast sensitivity is a simplification, but it allows the generation of simple predictions that 
capture the essential features of the data, and that are more than adequate in applied situations. These 
predictions follow from a reasonable conjecture. We hypothesize that two stimuli will appear identical if 
their spectra, after passing through the window of visibility, are identical 

A more precise expression of this hypothesis is that the spatiotemporal distribution of contrast in the 
image is filtered at some stage in the visual system. The limits of the pass-band of this filter are uj and vv^. If 
after passing through the filter two stimuli are identical, then an observer relying upon the output of this 
filter will be incapable of distinguishing between the two. 
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Figure 3.— Window of visibility. The shaded region contains combinations of spatial and temporal frequency that are invisible 
to the human eye. The window is bounded by uj and Wj, the limits of sensitivity to spatial and temporal frequency. 


CRITICAL SAMPLING FREQUENCY 


Note that the spectrum of the sampled line differs from that of the smooth line only by the addition of 
the parallel replicas at intervals of the sampling frequency. Thus the conjecture above implies that if these 
replicas lie outside the window of visibility, then the smoothly moving line and the sampled line will be 
indistinguishable. The replicas may be moved outside the window of visibility by either increasing the sam- 
pling frequency (which moves the replicas farther from the origin), or by reducing the velocity (which makes 
the replicas more nearly vertical). More precisely, note that for any velocity, the critical sampling frequency 
will be achieved when the first spectral replica is just touching the corner of the window of visibility, as 
shown in figure 4. The coordinates of this corner are (u^,u’^), the slope of the line impulse is , and it inter- 
sects the w axis at the point 0). From this information it is simple algebra to relate the sampling Ire- 
quency to r, uj, and Specifically, the critical sampling frequency, w^. at which smooth and sampled 
motions become indistinguishable is given by 


w^ = wi + rui (b) 

Thus the predicted critical sampling frequency is a linear function of velocity, witli an intercept given by the 
temporal frequency limit, and slope given by the spatial frequency limit. 


6 




Figure 4.— Boundary condition for identical appearance of smooth and stroboscopic motion. For clarity, only the first spectral 
replica on the right is shown. It is just touching the corner of the window of visibility. 


EXPERIMENT 1 


This prediction was tested by means of a two-interval forced-choice experiment. One interval contained 
a vertical line which moved smoothly to the right or left, the other interval contained a line moving at the 
same velocity but sampled at a rate of Tlie observer was asked to choose which interval contained the 
sampled version, and was informed after each trial whether the choice was correct. The smooth line was in 
fact sampled at 1920 Hz. This is effectively smooth, given the spatial and temporal transfer properties of the 
cathode ray display. The stimulus was a vertical line 50 min of arc in length and 0.65 min wide which moved 
horizontally at the specified velocity. Observers fixated a point at the center of the path of travel. The dis- 
tance traveled wasiyF 5/4 deg, and the duration 5/(4,/?) sec. Viewing was binocular with natural pupils from 
a distance of 2 m. Both observers were corrected myopes. Background luminance was 50 cd m’^. Stimuli 
were generated on an Evans and Sutherland PSl caligraphic display. The spatial contrast of each sample in 
the smooth line was 200%. in the sampled line (1920/w^) 200%, so that the two versions were equated for 
time-average contrast. The order of presentation was randomized on each trial, and the direction of motion 
was randomized on each presentation. A session consisted of 25 trials at each of five sampling frequencies, 
all at a single velocity. From the frequency of correct responses as a function of sampling frequency, the 
critical sampling frequency was estimated at which the observer was correct 75% of the time. 

Figure 5 shows the estimates of critical sampling frequency as a function of velocity for two observers. 
In each case the critical sampling frequency increases approximately linearly with velocity, as predicted by 
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on as a function of velocity for two observers. The straiglit lines 
intercept {wj) of each line is indicated. 



equation (6). For both observers the intercept is at about 30 Hz, which is a good estimate for the temporal 
frequency limit (wj) under these conditions. The slope of the curve, which according to theory is an estimate 
of the spatial frequency limit (up, is 6 cycles/deg for one observer and 13 cycles/deg for the other. These are 
somewhat low for estimates of the spatial frequency limit, but are not unreasonable given the low contrast 
and brief duration of the frequency component presumably serving to distinguish between smooth and 
sampled versions. Thus the data in figure 5 support the hypothesis that smooth and sampled motion are visu- 
ally indistinguishable when the spectral components that differ between them lie outside the window of 
visibility. 

To make a more precise prediction, it is necessary to know the bandwidth of the detector (or detectors) 
that discriminate between the smooth and the sampled motions. Without this information, the required con- 
trast of the line cannot be derived from the contrast sensitivity to a sinusoidal grating. For example, a 
detector of 1 -octave bandwidth (Watson, 1982, 1983) centered at 10 cycles/deg will respond equally to the 
first spectral replica of the line at 200% contrast, and to a sinusoidal grating with 37% contrast. Quantitative 
predictions would also have to take into account the detailed shape of the window of visibility, the duration 
of the stimulus, the inhomogeneity of spatial sensitivity across the retina, and possible masking by the spec- 
tral components lying within the window of visibility. Such predictions can be made, but are beyond the 
scope of this report. 


CONTRAST DISTRIBUTION OF STAIRCASE MOTION 


Another effective stimulus for apparent motion is called 2 i staircase presentation because of the appear- 
ance of its graph of position with respect to time. It differs from stroboscopic motion in that each presenta- 
tion lasts the full interval between steps. Since this method of presentation is often used and discussed in the 
literature on apparent motion, it was of interest to discover whether the window of visibility theory could be 
applied to it as well. 

The contrast distribution of staircase motion is derived by first constructing a function representing one 
■‘stair” of the staircase 


z{x,t)-w^u{tw^)b{x) (7) 

where u{t) is the unit pulse function. The stair function is pictured in figure 6(a). The full staircase is con- 
structed by convolving the stair function with the strobe function constructed earlier 

l^{x,t) = l^{x,t)*z{x,t) (8) 


This function is pictured in figure 6(b). 


FREQUENCY SPECTRUM OF STAIRCASE MOTION 


To get the transform, the convolution theorem is again applied. 

L^{u,w} = L^(u,w)Z{u,w) (9) 

L^(u,w) in equation (5) and figure 1(d) have already been determined and reproduced in figure 6(c). The 
transform of the stair is 
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TIME, sec 




TEMPORAL FREQUENCY, Hz 


,urc 6.— Derivation of the frequency spectrum of staircase motion, (a) The “stair” function is a unit pulse in t multiplied by 
an impulse in x. (b) The contrast distribution of staircase motion is the convolution of the stair function with the strobo- 
scopic motion function pictured in figure 1(b). (c) Frequency spectrum of the stroboscopic motion function, (d) Fre- 
quency spectrum of the stair function, a sine function with its first zero at uv. 




( 10 ) 


Z(m,w) = Fr^ w^m(w^05(x)[ 

= sinc{w/w^) 

This function is pictured in figure 6(d). The transform is the product of Z(u,w) and L^{u,w) which is illus- 
trated in figure 7. It differs from that for stroboscopic motion in that each line impulse is “shaved off’ by 
the sine function, falling to its first zero at w^. 

When will the staircase motion be just indistinguishable from smooth motion? As in the case of strobo- 
scopic motion, the replicas must be kept outside the window of visibility. This leads to the same sampling 
requirement specified for stroboscopic motion by equation 6. But when this condition is met smooth and 
sampled spectra still differ by the portion of the central line shaved off by the sine function (fig. 8). This 
difference (indicated by stipling in fig. 8) is never more than 12% of the total spectrum, and is usually much 
less. Furthermore, this difference lies in a region in which sensitivity within the window is low. It therefore 
seems unlikely that critical sampling frequency for staircase motion should differ much from that for strobo- 
scopic motion. 

EXPERIMENT 2 


To test this prediction, experiment 1 was repeated for stroboscopic and staircase motion. For staircase 
motion the line was presented for the full interval between samples (At). The stroboscopic case was repeated 
because thresholds were collected by a method of adjustment, rather than by the forced-choice method used 
in experiment 1, In the adjustment method, the observer was presented with a sequence of alternating 



Figure 7.- Frequency spectrum of staircase motion, L,(u,w). The modulus of the spectrum is shown for clarity. 
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Figure 8.— Windowed spectra for stroboscopic and staircase motion when the sampling frequency is given by = wj + nij. The 
stipled region indicates the difference between the two. The support plane is commensurate with the window of visibility. 


smooth and sampled motions, and was asked to adjust the sampling frequency until tlie two appeared just 
discriminable. 

The results for two observers are shown in figure 9. The important observation is as predicted that 
staircase and stroboscopic presentation require the same critical sampling rate. The stroboscopic data col- 
lected with method of adjustment are very similar to the forced-choice data of experiment 1. 


SPATIAL DEPENDENCE OF THE CRITICAL SAMPLING FREQUENCY 


The spatial stimulus thus far considered is a narrow line that has spatial frequencies extending well 
beyond the window of visibility. When the stimulus contains a restricted range of spatial frequencies, the pre- 
dictions change somewhat. Consider the case of a stimulus band-limited to below Uq cycles/deg. The spec- 
trum will again lie along a line with a slope of r"* , but it will terminate at Uq and -Uq. When this stimulus is 
presented stroboscopically at the critical sampling frequency, the situation diagrammed in figure 10 will 
result. The first replica just touches the window when 

vv^ = wj + niQ (11) 
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SAMPLING FREQUENCY, Hz SAMPLING FREQUENCY, Hz 



VELQCITY, deg/sec 

Figure 9.- Critical sampling frequency as a function of velocity for staircase and stroboscopic motion. The dashed line is a 
least squares fit to the stroboscopic data (ABW: intercept = 33.2, slope = 17.0; JEF: intercept = 46.2, slope = 1 1.0). 
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Figure 10.— Boundary condition for a moving stimulus spatially band-limited to below mq cycles/deg. The slope of the spec- 
trum is r'‘ . The first replica is just touching the window of visibility at the point vv/wq. 


Note that this situation is the same as that for equation 6, except that the spatial border of the window 
has been replaced by the spatial border of the stimulus Uq. It therefore seems appropriate to generalize and 
say that the spatial frequency term in equation (11) should be regarded as the highest “effective” spatial 
frequency in the stimulus. This quantity will be given by the limit of the window or the stimulus, whichever 
is lower. 


EXPERIMENT 3 


This prediction was tested by asking observers to distinguish between two vertical sinusoidal gratings 
which drifted at the same rate (one effectively smooth and the other sampled at some rate). The use of 
gratings allows particularly simple predictions, since the critical sampling frequency should be equal to the 
temporal frequency limit plus the velocity times the spatial frequency of the grating. 

The gratings were presented at a 20% contrast on a 50 cdm'^ background (P-31 phosphor). Display 
frame rate was 200 Hz, so sampling frequencies were limited to integral divisors of this rate (100, 66.7, 50, 
40, 33.3, 28.6, and 25 Hz), thus limiting the range of velocities that could be examined and the accuracy 
with which critical frequency could be estimated. Otherwise, conditions were similar to those in experi- 
ment 1. These results are shown in figure 1 1(a) and (b). Figure 1 1(b) shows data for a grating of 1 cycle/deg. 
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Between 0 and 8 deg/sec the sample frequency rises by about 6 Hz, close to the predicted value of 8 Hz. 
Figure 1 1(a) shows data for 4 cycles/deg. Between 0 and 8 cycles/deg, the sample frequency rises by about 
25 Hz, close to the predicted value of 32 Hz. 


RELATION TO APPARENT MOTION 


It has been shown that stroboscopic and staircase apparent motion, in which a long sequence of many 
views is presented to the observer, are explained by the spatiotemporal filtering action of the eye. These two 
cases constitute the most compelling varieties of apparent motion. However, many classic instances of appar- 
ent motion use only two samples, or two samples in repeated alternation. In such displays the illusion has 
been reported to occur over distances of several degrees and time intervals of several hundred milliseconds, 
well outside the limits for perfect fidelity discovered here (Kolers, 1972). But two-sample displays evidently 
produce an illusion much inferior to that obtained with many samples (Sperling, 1976). It remains to be seen 
whether such displays are indistinguishable from a corresponding real motion, and whether their appearance 
can be explained by the theory presented here. It may be possible, however, that after passage through the 
visual band-pass filter discussed above, such stimuli are no longer discontinuous in space or time. 

Morgan (1979, 1980a, 1980b) has also proposed a filter theory of apparent motion, but it takes as 
input the function relating displacement to time, rather than the function relating contrast to space and 
time. His filter is therefore purely temporal, and does not predict the relation between critical sampling rate, 
velocity, and spatial frequency discovered here. 


IMAGE RECORDING 


Many images that appear on stroboscopic displays were recorded by a camera. The camera recording 
process acts as a temporal filter and thus reduces the sampling rate required in subsequent display. The filter- 
ing action occurs either because the aperture is left open for a while during each frame or because the react- 
ing medium (film or video tube) have finite reaction times. 

To see the effect of this filtering, assume that the recording process removes all temporal frequencies 
above wy. When an image moves with velocity r, its spectrum tilts in the u-w plane with slope r”' . Thus, as 
shown in figure 12, all spatial frequencies above u = are removed. Substituting this as the highest effec- 

tive spatial frequency Uq in equation 1 1, gives 

Wg = Wi + r{w^r^ ) 

= \\>i + ( 12 ) 

This condition will hold whenever is less than Uq or u^. Thus the sampling requirements remain con- 

stant, regardless of the velocity. In effect, the amount of spatial filtering just compensates for the increased 
sampling rate that an increase in velocity would otherwise require. Furthermore, the spatial filtering occurs 
only when the image moves, so that stationary images can be viewed with high detail. 

All of the cases considered above can now be summarized in one equation, in which the Iiighest effec- 
tive spatial frequency is given by the least of the three possible limits 
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= wi + r min(uQ,Ui,Wj-r * ) 


(13) 


Wj 



Figure 12.- The effect of temporal filtering upon the spectrum of a moving image. The band limit of the temporal filter is 

The temporal filter removes all spatial frequencies above . 

COMPUTER-GENERATED IMAGERY 


Computer-generated images bypass the camera recording process, and are not subject to the spatial and 
temporal prefiltering described above. In “object space,” that is the coordinate space in which the image is 
defined internal to the computer, the image may have infinitely high spatial and temporal frequencies. It is 
for this reason that presentation of computer-generated imagery on conventional television displays, with 
their sampling frequency of 30 or 60 Hz, often gives rise to serious artifacts. 

One possible solution to this problem is to simulate the recording process in the computer. This might 
be possible by sampling the scene at extra-high resolution, averaging the last n frames, and then sampling at 
the resolution of the display. However, this would require that all computations necessary to get from object 
space to image space (projection, hidden-line removal, surface generation, shading, etc.) be done at the extra- 
high resolution. An alternative strategy would be to code the image in spatial frequency bands and then 
select for display only those bands that the velocity and sampling frequency will not alias. 


CONCLUSIONS 


The general notions presented here regarding sampled displays and visual filtering can be extended to an 
arbitrary spatial image undergoing an arbitrary transformation over time, and the sampling process can be 
extended to the two spatial dimensions as well as to time. They provide answers to some long-standing 
puzzles in perceptual psychology, and to some modem problems in advanced visual displays. 


Ames Research Center 

National Aeronautics and Space Administration 

Moffett Field, California 94035, February 18, 1983 
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